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Sir: 

-This is in response to the Examiner's non- final Office Action of August 31, 2001 . 
Please amend the above-identified application as follows: 

IN THE SPECIFICATION: 



Please substitute the attached specification incorporating all of the changes made by 
Certificate of Correction as required by the Examiner in paragraph 6 of the Office Action of 
August 31, 2001. 

IN THE CLAIMS: 



SYSTEMS 



AMENDMENT 




Please amend claim 25 as folkn 
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25. (Twice Amended) A display system comprising: 
a first backend pipeline for processing data; 

a second backend pipeline for processing graphics data disposed in parallel to said 
first processing pipeline; 

a multi-format frame buffer memory having on-screen and off-screen areas each 
operable to allow said frame buffer to simultaneously store data in graphics and video 
formats; 

a input port for receiving both graphics and video data, each word of said data 
associated with an address directing said word to be processed as either graphics or video 

y3 data; 

O 

^ circuitry for writing a word of said graphics and video data into a selected one of 

said areas of said multi-format memory; 

4= 

1^ memory control circuitry for controlling the transfer of data between said first 

backend pipeline and said frame buffer and between said second backend pipeline and 
O said frame buffer; 

!r 

p a display unit; and 

overlay control circuitry for selecting for output to said display unit between data 
provided by said first backend pipeline and data provided by said second backend 
pipeline. 
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REMARKS 

Claims 1-48 remain in the application. Claims 13-24 and 34-36 stand allowed. 
Claims 42, 44 and 48 stand objected to but are held to distinguish over the prior art. Claims 
1-12, 25-33, 37-41, 43 and 45-47 stand rejected. The undersigned appreciates the 
Examiner's rinding that the Bindlish reference was overcome by the Declaration under 37 
CFR 1.131. 

The Examiner indicated that the original Patent or an Affidavit or Declaration as to 
the loss or inaccessibility of the original patent must be received before the reissue 
application can be allowed. The undersigned has requested the Assignee to provide the 
original patent or to execute an Affidavit or Declaration as to its loss or inaccessibility. 
Applicants are attempting to determine the status of the original patent and will provide an 
appropriate response before the reissue application can be allowed. Applicants therefore 
respectfully request that compliance with this requirement be deferred until such time as the 
information becomes available. 

The Examiner has requested a clean copy of the specification with the Certificate of 
Correction changes incorporated therein. A copy of such a clean specification is attached 
hereto. 

The Examiner rejected claims 1-12 under 35 USC 112, first paragraph: 

"as containing subject matter which was not described in the 
specification in such a way as to reasonably convey to one 
skilled in the relevant art that the inventor(s), at the time the 
application was filed, had possession of the claimed 
invention. Claim 1 recites "an interface for receiving words 
of pixel data, each said word associated with an address 
buffer". There is no clear written description in the 
specification of "an address buffer". 
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The claim language for which the Examiner finds no written description is contained 
in claim 1 as it was originally filed. In addition, the specification provides ample support for 
such a language. See, for example, page 12, line 30 to page 14, line 10. See also page 11, 
line 2 through page 11, line 29 (and especially lines 12-22). There are also other places 
throughout the specification which provide more than adequate written description for the 
claim language in question. Accordingly, since the claim language in question was part of 
the original claims and is described in the original specification, Applicants respectfully 
request that the Examiner withdraw the rejection under 35 USC 1 12, first paragraph. 

The Examiner rejected claims 25-33 under 35 USC 112, second paragraph, as 
indefinite because the limitation "said playback data" in column 17, line 42 has "insufficient 
antecedent basis." Claims 26-33 are dependent upon claim 25 and the Examiner therefore 
considered them to similarly lack antecedent basis. Applicants have amended claim 25 to 
ensure that the claim is definite and has adequate antecedent basis. 

The Examiner rejected claims 37-39 and 41 under 35 USC 102(e) as anticipated by 
U.S. Patent No. 5,406,306 to Siann et al. 
« The Examiner rejected claims 1-6 and 12 under 35 USC 103 as unpatentable over 

Siann et al. in view of Herbert and Gery. 

The Examiner rejected claims 40, 43 and 45-47 under 35 USC 103 as unpatentable 
over Siann et al. in view of Gery. 

Each of these rejections is predicated upon the Siann et al. reference. 
The Siann et al. reference was discussed in the public version of the Initial 
Determination by the presiding Administrative Law Judge for the International Trade 
Commission in investigation number 337-TA-412 entitled In the Matter of Certain Video 
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Graphics Display Controllers and Products Containing Same. A discussion of the Siann et 
al. reference occurs on pages 95 and 96 of that document. Summarizing, the staff of the 
International Trade Commission, the presiding Administrative Law Judge of the 
International Trade Commission and the expert witness for ATI Technologies, Inc. all 
determined that the Siann et al. reference does not have circuitry for "selectively retrieving" 
as required by claim 37. This finding was not reversed by the entire Commission. 

This is directly contrary to the Examiner's finding on page 4 of the Office Action 
that "Siann teaches 'circuitry for selectively retrieving' at col. 4 lines 59-col. 5 line 13." 

The portion of Siann et al. referred to by the Examiner reads as follows: 

"The display memory 52 may have a first portion 54 for 
storing graphics information and a second portion 56 for 
storing video information. The graphics information stored 
in the portion 54 of the display memory 52 provides the data 
for the pixels on the face 10 of the video monitor 12 outside 
of the window 14 and also provides coded data signals at the 
pixel positions within the window 14 for insuring that the 
video information in the portion 56 of the display memory 52 
will be displayed in the window. 

The video information stored in the portion 56 of the display 
memory 52 provides input data to generate the color pixels in 
the window 14 on the face 10 of the video monitor 12. The 
video information stored in binary form in the portion 56 
may indicate the luminance and the two (2) quadrature 
components of chrominance for each pixel. The video 
information stored in binary form in the portion 56 of the 
display memory 52 may be in a compressed form which may 
be decompressed to provide 320 video pixels in each of 240 
lines. The graphics information stored in the portion 54 of 
the memory 52 may be in the form of 1024 pixels stored for 
each of the 768 lines." 



A consideration of the words of the portion of the Siann et al. patent referred to by 
the Examiner does not show any teaching of "circuitry for selectively retrieving" as required 
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by the claim language. Further, the Examiner's position is expressly contradicted by the 
findings of the ITC staff, the ITC presiding Administrative Patent Judge and ATI's own 
expert. 

The Examiner's findings cannot ignore the determinations of the courts nor of the 
experts. In reMeng, 492 F2d 843, 181 USPQ 94 (CCPA 1974). 

Further, if the Examiner is attempting to rely on a doctrine of inherency, inherency 
may not be established by probabilities or possibilities. The mere fact that a certain thing 
may result from a given set of circumstances is not sufficient. Hansgrig v. Kemmer, 40 
USPQ 667 (CCPA 1939); In re Oelrich, 666 F2d 578, 581, 212 USPQ 323, 326 (CCPA 
£ 1981). 

O Arguments other than those expressly set forth above were asserted during related 

P 

'£ litigations. The fact that some of those arguments are not repeated at this time does not 
mean that those other arguments are waived or are not significant. Rather, the arguments 
y, expressly asserted in this response are considered sufficient to overcome the Examiner's 

b 

3 rejections. 

Q In the face of express findings by the court, Applicants respectfully request that the 

Examiner reconsider the holding that Siann et al. teaches "circuitry for selectively 
retrieving" as required by the claims. Independent claims 1, 37 and 43 each require this 
limitation. These are all of the independent claims under rejection by the Examiner on prior 
art. Since Siann et al. do not teach or suggest limitation of the claims under rejection, 
Applicants respectfully request the Examiner to withdraw the rejection and permit the 
application to issue as a patent. 
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For the reasons given, Applicants believe that the application is in condition for 
allowance and Applicants request that the Examiner give the application favorable 
consideration and permit it to issue as a patent. 

Please charge any shortage in fees due in connection with the filing of this paper, 
including extension of time fees, to Deposit Account 500417 and please credit any excess 
fees to such deposit account. 



Respectfully submitted, 



MCDERMOTT, WILL & EMERY 





David L. Stewart 
Registration No. 37,578 



If! 600 13 th Street, N.W. 

£ Washington, DC 20005-3096 

M (202) 756-8000 DLS:kap 

^ Date: October 1, 2001 



Facsimile: (202) 756-8087 
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VERSION WITH MM^fTO SHOW CHANGED MADE 

IN THE SPECIFICATION: 

Please see the attached substitute specification, which is a clean copy of the 
specification incorporating all of the changes made by Certificate of Corrections of U.S. 
Patent No. 5,598,525. 

IN THE CLAIMS: / 



Claim 25 has been amended as follows: 

25. (Twice Amended) A display system comprising: 
a first backend pipeline for processing data; 

a second backend pipeline for processing graphics data disposed in parallel to said 
first processing pipeline; 

a multi-format frame buffer memory having on-screen and off-screen areas each 
operable to allow said frame buffer to simultaneously store data in graphics and video 
formats; 

a input port for receiving both graphics and video data, each word of said data 
associated with an address directing said word to be processed as either graphics or video 
data; 

circuitry for writing a word of said [playback] graphics and video data into a 
selected one of said areas of said multi-format memory; 

memory control circuitry for controlling the transfer of data between said first 
backend pipeline and said frame buffer and between said second backend pipeline and 
said frame buffer; 

a display unit; and 

overlay control circuitry for selecting for output to said display unit between data 
provided by said first backend pipeline and data provided by said second backend 
pipeline. 
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APPARATUS. SYSTEMS AND 
METHODS FOR CONTROLLING 
GRAPHICS AND VIDEO DATA IN 
MULTIMEDIA DATA PROCESSING 
AND DISPLAY SYSTEMS 



TECHNICAL FIELD OF THE INVENTION 

The present invention relates in general to multimedia processing and 
display systems and in particular to apparatus, systems and methods for 
5 controlling graphics and video data overlay in multimedia processing and display 
systems. 

CROSS-REFERENCE TO RELATED APPLICATIONS 

The following copending and coassigned United States patent applications 
contain related information and are incorporated herein by reference: 
10 U.S. patent application Ser. No. 08/098,846 (Attorney's Docket No. 

P3510-P11US), entitled "System And Method For The Mixing Of Graphics And 
Video Signals," and filed Jul. 29, 1993; and 

U.S. patent application Ser. No. 08/223,845 (Attorney's Docket No. 
P3510-P21US), entitled "Apparatus, Systems And Methods For Processing Video 
15 Data In Conjunction With A Multi-Format Frame Buffer," and filed Apr. 6, 1994. 

BACKGROUND OF THE INVENTION 

As multimedia information processing systems increase in popularity, 
system designers must consider new techniques for controlling the processing and 
20 display of data simultaneously generated by multiple sources. In particular, there 
has been substantial demand for processing systems which have the capability of 
concurrently displaying both video and graphics data on a single display screen. 
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The development of such systems presents a number of design challenges, not 
only because the format differences between graphics and video data must be 
accounted for, but also because of end user driven requirements that these systems 
allow for flexible manipulation of the data on the display screen. 
5 One particular technique for simultaneously displaying video and graphics 

data on a single display screen involves the generation of "windows." In this case, 
a stream of data from a selected source is used to generate a display within a 
particular region or "window" of the display screen to the exclusion of any non- 
selected data streams defining a display or part of a display corresponding to the 
10 same region of the screen. The selected data stream generating the display 
window "overlays" or "occludes" the data from the nonselected data streams 

f which lie "behind" the displayed data. In one instance, the overall content and 
-v ^ appearance of the display screen is defined by graphics data and one or more 

J h£ "video windows" generated by data from a video source occlude a corresponding 

1 5 region of that graphics data. In other instances, a video display or window may be 
^ occluded or overlaid by graphics data or even another video window, 

f t In the multimedia environment, the "windowing" described above yields 

□ substantial advantages. Among other things, the user can typically change the size 

iJ, and location on the display screen of a given window to flexibly manipulate the 

p 20 content and appearance of the data being displayed. For example, in the case of 
combined graphics and video, the user can advantageously create custom 
composite visual displays by combining multiple video and graphics data streams 
in windowing environment. 

In order to efficiently control windows in a multimedia environment 
25 efficient frame buffer management is required. Specifically, a frame buffer 
control scheme must be developed which allows for the efficient storage and 
retrieval of multiple types of data, such as video data and graphics data. To be 
cost competitive as well as functionally efficient, such a scheme should minimize 
the number of memory devices and the amount of control circuitry required and 
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should insure that data flow to the display is subjected to minimal delay 
notwithstanding data type. 

One of the major difficulties in managing video in a combined video and 
graphics windowing environment results from the fact that the video data being 
5 received and displayed are constantly being updated, typically at a rate of thirty 
frames per second. In contrast, the graphics data are normally generated once to 
define the graphics display and then remain static until the system CPU change 
that graphics display. Thus, the occlusion (overlay) of video data with graphics 
data requires that the static graphics data "in front of the video data not be 
10 destroyed each time the video window is updated. A second concern with 
windowing systems operating on both video and graphics data is the formatting 
s 5 differences between the video and graphics data themselves since video is 

"jf typically digitized into a YUV color space while graphics is digitized into an 

IS RGB color space. Hence, any combination video and graphics windowing system 

1 15 must have the capability of efficiently handling data within both the YUV and 
M. RGB formats. 

s Thus, due to the advantages of windowing, the need has arisen for 

O efficient and cost effective windowing control circuitry. Such windowing circuitry 

2 should allow for the simultaneous processing of data received from multiple 
P 20 sources and in multiple formats. In particular, such windowing control circuitry 

should be capable of efficiently and inexpensively controlling the occlusion 
and/or overlay of video and graphics data in a windowing environment. 



SUMMARY OF THE INVENTION 
25 The principles of the present invention in general provide for the flexible 

control of graphics and video data in a display control environment. In particular, 
an entire frame of video data, graphics data, or a combination of both, may be 
stored in on-screen memory and rastered out with the generation of the 
corresponding display screen. A window of graphics or video data can then be 
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stored in off-screen memory and retrieved when the raster scan generating the 
display reaches the desired position on the display for the video window. The 
window of data from off-screen memory can then be overlayed over the data 
being rastered out of the on-screen memory under one of three conditions. In a 
5 first mode, pixels from the off-screen memory are rastered only when the raster 
scan has reached the position on the display selected for the window. In a second 
mode, a window of data is rastered from the off-screen memory when the display 
raster scan has reached the display window position and graphics data being 
rastered from the on-screen memory matches a color key. In a third mode, the 
10 window data is rastered out of the off-screen memory when the data being output 
from the on-screen memory matches the color key, notwithstanding the position 
dfj of the raster scan. 

g According to a first embodiment of the present invention, a graphics and 

O video controller is provided which includes a dual aperture interface, each word 

£ 15 associated with an address to a selected one of on-screen and off-screen areas of 
J, an associated unified frame buffer as either graphics or video pixel data. Circuitry 

. ^ is provided for writing a word of the pixel data received by the interface to a one 

p of the on-screen and off-screen memory areas corresponding to the address 

W associated with the received word. Circuitry is also included for selectively 

p 20 retrieving graphics and video data from the on-screen and off-screen memory 
areas. A first pipeline is provided for processing graphics data retrieved from the 
frame buffer and a second pipeline is provided for video processing data retrieved 
from the frame buffer. 

According to a second embodiment of the present invention, a controller is 
25 provided which includes a dual aperture port for receiving video and graphics 
data, each word of the data received with an address associated directing the word 
to be processed as either graphics or video data and off-screen memory spaces of 
a frame buffer. A second port is included for receiving real-time video data. 
Circuitry is provided for generating an address associated with a selected one of 
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the memory spaces for each word of received real-time video data. Circuitry is 
included for writing selectively the words into the on-screen and off-screen 
memory spaces of the frame buffer. Circuitry is also provided for selectively 
retrieving the words of data from the on-screen and off-screen spaces as data is 
rastered for driving a display. A graphics backend pipeline processes ones of the 
graphics words of data retrieved from the frame buffer. A video backend pipeline 
is provided for processing ones of the video words of data retrieved from the 
frame buffer, the circuitry for retrieving always rastering a stream of graphics data 
from the frame buffer to the graphics pipeline and rastering video data to the 
video backend pipeline when a display raster scan reaches a display position of a 
video window. An output selector is included for selecting for output between 
words of data output from the graphics backend pipeline and words of data output 
from the video backend pipeline, 
fl According to a third embodiment of the present invention, a display 

=P 15 system is provided which includes first and second parallel backend pipelines. A 
§2 multi-format frame buffer memory is included having on-screen and off-screen 

jj\ memories each operable to simultaneously store data in graphics and video 

O formats. A dual aperture port is provided for receiving both graphics and video 
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data as directed by an address associated with each word of data received. 
Circuitry for writing is included for writing a word of video or graphics data into 
a selected one of the on-screen and off-screen areas of the multi-frame buffer. 
Memory control circuitry controls the transfer of data between the first and 
second backend pipelines and the frame buffer. The system further includes a 
display unit and overlay control circuitry for selecting for output to the display 
unit between data provided by the first backend pipeline and data provided by the 
second backend pipeline. 

A fourth embodiment of the present invention comprises a display data 
processing system which includes circuitry for writing data into an on-screen 
space of a frame buffer and circuitry for writing data into an off-screen space of 
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the frame buffer. A video pipeline is provided for processing video data output 
from a selected one of the on-screen and off-screen spaces. The video pipeline 
includes a first first-in/first-out memory for receiving selected pixel data from the 
selected space. The video pipeline also includes a second first-in/first-out memory 
disposed in parallel to the first first-in/first-out memory for receiving other 
selected data from the selected space in the frame buffer. An interpolator is 
provided as part of the video pipeline for generating additional data by 
interpolating data output from the first and second first-in/first-out memories. A 
graphics pipeline is disposed in parallel to the video pipeline for processing 
graphics data output from a selected one of the on-screen and off-screen spaces. 
*Y Finally, an output selector is provided for selecting between data output from the 

video pipeline and data output from the graphics pipeline. 

The principles of the present invention allow for the construction of 
circuits and systems with substantial advantages over the prior art. Among other 
JE 15 things, the principles of the present invention allow both graphics and video data 
to be stored in a single unified frame buffer and retrieved therefrom in a number 
s of different ways. For example, a combination of graphics and video data may be 

P stored in the on-screen memory and simply rastered out during screen refresh. In 

O another case, an entire screen of graphics or video data may be stored in the on- 

O 20 screen memory while a window of graphics or video data is stored in the off- 
~ screen portion of memory. The window data can then be rastered out to 

selectively overlay a portion of the data being rastered out of the on-screen 
memory. The overlay may be controlled by either window display position with a 
match of the on-screen data being rastered out and a color key, or both. 
25 The embodiments of the present invention provide for the efficient and 

inexpensive overlay of video and graphics data in a windowing environment. In 
particular, the use of color comparison to determine the overlay of data in a 
window region eliminates the need for precise x- and y-position data for the 
location of that window and allows for video cropping to be performed. Further, 
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the use of graphics data to control overlay provides substantial advantages in that 
graphics data is less subject to the graininess and noise problems often found with 
video data. Further, the user is given total control of overlay operations when 
keying on graphics data because the graphics data is computer generated, whereas 
5 the video data is captured data. 

BRIEF DESCRIPTION OF THE DRAWINGS 

For a more complete understanding of the present invention, and the 
advantages thereof, reference is now made to the following descriptions taken in 
) conjunction with the accompanying drawings, in which: 

FIG. 1 is a top level functional block diagram of a multi-media processing 
and display system embodying the principles of the present invention; 

FIG. 2 is a more detailed functional block diagram of the VGA controller 
depicted in FIG. 1; 

J+j 15 FIG. 3 is an expanded functional block diagram of portions of the 

|=i controller of FIG. 2 with emphasis on the overlay control features; 

^ FIG. 4A is a detailed functional block diagram of a first embodiment of 

S the color comparison circuitry of FIG. 3; 

jU FIG. 4B is a detailed functional block diagram of a second embodiment of 

r 20 the color comparison circuitry of FIG. 3; and 

FIG. 5 is a detailed functional block diagram of a selected one of the video 
window position control circuits depicted in FIG. 3. 

DESCRIPTION OF THE INVENTION 
25 FIG. 1 is a high level functional block diagram of a multi-media 

processing and display system 100 operable to process and simultaneously display 
on a single display screen both graphics and video data according to the principles 
of the present invention. Display system 100 includes a central processing unit 
(CPU) 101 which controls the overall operation of system 100 and generates 
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graphics data defining graphics images to be displayed. CPU 101 communicates 
with the remainder of the system discussed below via a local bus 103. System 100 
also includes a real-time video data source 104. A real time video stream may be 
presented to the system VGA controller 105 in one of two ways. First, video data 
source 104 may be coupled to local bus 103 and a video data stream introduced 
through dual aperture addresses. In this case, video source 104 will directly 
address the system frame buffer 107. Second, video source 104 may be coupled 
directly to VGA controller 105 via a dedicated bus 109 or "video port." In this 
instance, VGA controller 105 generates the required addresses into frame buffer 
107. Real-time video source 104 may be, for example, a CD ROM unit, a laser 
disk unit, a videotape unit, television cable outlet or other video data source 
outputting video data in a YUV format. CPU 101 operates in conjunction with a 
system memory 108 which stores graphics and video data on a real-time basis. 
System memory 108 may be for example random access memory (RAM), floppy 
disk, hard disk or other type of storage device. 

A VGA controller 105 embodying the principles of the present invention 
is also coupled to local bus 103. VGA controller 105 will be discussed in detail 
below; however, VGA controller 105 generally interfaces CPU 101 and video 
source 104 with a display unit 106 and a multiformat system frame buffer 107. 
Frame buffer memory 107 provides temporary storage of the graphics and video 
data during processing prior to display on display unit 106. According to the 
principles of the present invention, VGA controller is operable in selected modes 
to store graphics and video data together in frame buffer 107 in their native 
formats. In a preferred embodiment, the frame buffer area is partitioned into on- 
screen memory and off-screen memory. Frame buffer 107 is also a "unified" 
memory in which video or graphics data can be stored in either the on-screen or 
off-screen areas. In the preferred embodiment, display unit 106 is a conventional 
raster scan display device and frame buffer 107 is constructed from dynamic 
random access memory devices (DRAMs). 
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FIG. 2 is a more detailed functional block diagram of VGA controller 105. 
The primary circuitry blocks of VGA controller 105 include video front-end video 
pipeline 200, memory (frame buffer) control circuitry 201, CRT/window control 
circuitry 202, video window control registers 203, video backend pipeline 204 
and graphics backend pipeline 205. VGA controller 105 further includes a CPU 
interface 206 for exchanging instructions and data via a PCI or VL bus, such as 
local bus 103 in system 100, with CPU 101. A write buffer 207 and conventional 
graphics controller 208 allow CPU 101 to directly control data within frame 
buffer 107 via memory control circuitry 201. 

In the preferred embodiment of system 100, CPU 101 can write video data 
and/or read and write graphics data to frame buffer 107 via CPU interface 206. In 
particular, CPU 101 can direct each pixel to the frame buffer using one of two 
maps depending on whether that pixel is a video pixel or a graphics pixel. In the 
preferred embodiment, each word of pixel data ("pixel") is associated with one of 
two addresses, one which directs interpolation of the pixel as a video pixel 
through video front-end pipeline 200 and the other which directs interpolation of 
the pixel as a graphics pixel through write buffer 207 and graphics controller 208. 
As a consequence, either video or graphics pixel data can then be input to CPU 
interface 206 from the PCI/VI bus through a single "dual aperture" port as a 
function of the selected address. 

Data which is input through the video port 211 is address-free. In this 
case, video window controls 213 generates the required addresses to either the on- 
screen memory area or the off-screen memory as a function of display location for 
the video window. In the preferred embodiment, window controls 213 generate 
addresses using the same video control registers 203 used to control retrieval of 
the video in the backend pipeline (i.e., the screen x and y position registers 500 
and 501 discussed below in conjunction with FIG. 5). When data is being 
received through both the CPU interface 206 and the VPORT 211 simultaneously, 
the data is interleaved into memory with the two write buffers 207 and 217 
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buffering the data such that neither stream is interrupted or forced into a wait state 
at the source component (i.e., bus 103 or video source 104). 

It should be noted at this point that frame buffer 107 includes at least two 
different data areas or spaces to which data can be directed by the given address 
(either CPU 103 or controls 213 generated). Each space can simultaneously store 
graphics or video data depending on the selected display configuration. The on- 
screen area corresponds to the display screen; each pixel rastered out of a given 
pixel location in the on-screen area defines a corresponding screen pixel. The off- 
screen area is used to store data defining a window for selectively overlaying the 
data from the on-screen memory, fonts and other data necessary by controller 105. 
Further, as will be discussed below, both graphics and video data may be rastered 
from frame buffer 107 and passed through video backend pipeline 204 while only 
graphics data is ever passed through graphics backend pipeline 205. 

According to the principles of the present invention, there are alternate 
ways of storing and retrieving graphics and video data from unified frame buffer 
107. 

For example, CPU 103 may write a static graphics background into part of 
the on-screen memory with the remaining "window" in the on-screen memory 
area filled with playback video data. "Playback" video data can be either (1) live 
video data input from the VPORT; (2) YUV (video) data written through interface 
206 by CPU 103; or (3) true color (5:5:5, 5:6:5, or 8:8:8) RGB graphics data (for 
example animation graphics data) written in through either the VPORT or 
interface 206. Similarly, a playback video background and a window of graphics 
data may be written into the on-screen area. In each of these cases, the data is 
rastered out as the display is without overlay; the video playback data is passed 
through the video backend pipeline 204 as a function of display position by 
controls 202 and the graphics data passed through the graphics backend pipeline 
250. 
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Windows of data retrieved from the off-screen memory can be retrieved 
and used to occlude a portion of the data being rastered out of the on-screen 
memory. For example, a window of playback data can be stored in the off-screen 
memory and a frame of static graphics data (either true color data or indices to 
5 CLUT 234) stored in the on-screen memory. In this case, the static graphics are 
rastered out of the on-screen memory without interruption and passed through the 
graphics backend pipeline 205. The window of data in the off-screen memory is 
rastered out only when the display position for the window has been reached .by 
the display raster and is passed through video backend pipeline 204. As discussed 
10 below, data from the video backend pipeline 204 can then be used to selectively 
occlude (overlay) the data being output from the graphics backend pipeline 205. 
*\ A window of static graphics data (true color or indices to the CLUT 234) can be 

!?f stored in off-screen memory and used to overlay playback video from the on- 

0 screen memory. The playback video data is passed through the video backend 



m 



pipeline 204 and the window of static graphics data is passed through the graphics 
^ backend pipeline 205. 

s Bit block transfer (BitBLT) circuitry 209 is provided to allow blocks of 

« graphics data within frame buffer 107 to be transferred, such as when a window 

U 

O of graphics data is moved on the display screen by a mouse. Digital-to-analog 

p 20 converter (DAC) circuitry 210 provides the requisite analog signals for driving 
^ display 106 in response to the receipt of either video data from video backend 

pipeline 204 or graphics data from backend pipeline 208. 

In implementing the operations discussed above, video front-end pipeline 
200 can receive data from two mutually exclusive input paths. First, in the 
25 "playback mode," playback (non-real time) data may be received via the PCI bus 
through CPU interface 206. Second, in the "overlay emulation mode" either real- 
time or playback video may be received through the video port interface 211 (in 
system 100 video port interface 211 is coupled to bus 109 when real-time data is 
being received). The selection of video from the PCI bus or video from video port 
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interface 21 1 is controlled by a multiplexer 212 under the control of bits stored in 
a video front-end pipeline control register within video control registers 203. In 
the playback mode, either CPU 101 or a PCI bus master controlling the PCI bus 
provides the frame buffer addresses allowing video front-end pipeline 200 to map 
data into the frame buffer separate and apart from the graphics data. In the overlay 
emulation mode, overlay input window controls 213 receives framing signals 
such as VSYNC and HSYNC, tracks these sync signals with counters to 
determine the start of each new frame and each new line, generates the required 
addresses for the real-time video to the frame buffer space using video window 
position data received from window controls 222 (as discussed above, in the 
preferred embodiment, video data is always retrieved from either the on-screen on 
off-screen memory and passed through video back-end pipeline 204 as a function 
of display position) and thus the position data from controls 222 is used to both 
write data to memory and retrieve data therefrom. In general, overlay input video 
control windows are controlled by the same registers which control the backend 
video pipeline 204, although the requisite counters and comparators are located 
internal to overlay input video control circuitry 213. 

Video front-end pipeline 200 also includes encoding circuitry 214 that is 
operable to truncate 16-bit YUV 422 data into an 8-bit format and then pack four 
such 8-bit encoded words into a single 32-bit word which is then written into the 
video frame buffer space of frame buffer 105. Conversion circuitry 215 is 
operable to convert RGB 555 data received from either the CPU interface 206 and 
the PCI bus or VPORT I/F 211 into YCrCb (YUV) data prior to encoding by 
encoding circuitry 214. Conversion circuitry 215 allows graphics data (for 
example in a 5:5:5 or 5:6:5 format) to be introduced through the VPORT or 
graphics data to be converted, packed and stored in a YUV format in the off- 
screen memory space by CPU 101. For a more complete description of encoder 
214 and the associated decoder 225 of video pipeline 204, reference is now made 
to incorporated copending coassigned application Ser. No. 08/223,845. The 
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selection and control of the encoding circuitry 214 and conversion circuitry 215 is 
implemented through multiplexing circuitries 212 and 216, each of which are 
controlled by bits in the video control registers. Finally, video front-end pipeline 
200 includes a write buffer/FIFO 217 which in one embodiment acts as a write 
5 buffer and in an alternate embodiment acts as a FIFO for the video backend 
pipeline 204. In embodiments where buffer 217 acts as a write buffer for then Y, 
zooming on the backend, as discussed below is by replication. In embodiments 
where buffer 217 operates as a FIFO, then the WORT and front and end color 
conversion by converter circuitry 215 are not used for writing data to frame buffer 
10 107. 

Memory control circuitry 201 includes an arbiter 218 and a memory 

*0 interface 219. Arbiter 218 prioritizes and sequences requests for access to frame 

O 

buffer 107 received from video front-end pipeline 200, graphics controller 208 
and bit block transfer circuitry 209. Arbiter 218 further sequences each of these 
?p 15 requests with the refresh of the display screen of display 106 under the control of 
CRT controller 202. Memory interface 219 controls the exchange of addresses, 
data, and control signals (such as RAS, CAS and read/write enable) to and from 
frame buffer 107. 

CRT control/video window control circuitry 202 includes the CRT 
20 controller 220, window arbiter 221, and video display window controls 222. CRT 
controller 202 controls the refresh of the screen of display 106 and in particular 
the rastering of data from frame buffer 107 to display unit 107 through DAC 210. 
In the preferred embodiment, CRT controller 220, through arbiter 218 and 
memory interface 219, maintains a constant stream of graphics data into graphics 
25 backend pipeline 205 from memory; video or playback graphics data is rastered 
out only when a window has been reached by the display raster as determined by 
display position controls of window controls 222 (see FIGS. 3 and 5 and 
accompanying text) and CRT controller 220. As will be discussed in further detail 
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below, the display of windows within the display according to the principles of 
the present invention is controlled in part by circuitry 202. 

Video backend pipeline 204 receives a window of graphics video data 
defining a display window from the on-screen or off-screen spaces in frame buffer 
5 107 through a pair of first-in/first-out memories 223 and 217 (in embodiments 
where buffer 217 is acting as FIFO B). In the preferred embodiment, each FIFO 
receives the data for every other display line of data being generated for display 
on the display screen. For example, for a pair of adjacent lines n-1 and n+1 in 
memory (although not necessarily adjacent on the display) for the display 
10 window, FIFO 223 receives the data defining window display line n-1 while FIFO 
224 receives the data defining window display line n+1. When buffer 217 is used 
as FIFO B, writes through video front end pipeline 200 are made through write 

O buffer I 207 and multiplexer 235. Alternatively, if buffer 217 is used as write 

O 

t m buffer II, then FIFO B is not implemented and only a single stream is processed 

r" t 15 by video backend pipeline 204 (no Y interpolation is performed and Y expansion 
ih* is by replication). As will be discussed further below (assuming both FIFO A and 

y, FIFO B are being used), one or more display lines, which falls between line n-1 

and line n+1, may be selectively generated by interpolation. Decoder circuitry 225 

Q 

M. receives two 32-bit packed words (as encoded by encoder 214), one from each 

rf 20 adjacent scan line in memory, from FIFOs 223 and 217. Each 32-bit word, which 
represents four YCrCb pixels, is expanded and error diffused by decoder 225 into 
four 16-bit YCrCb pixels. In modes where video data is stored in the frame buffer 
in standard 555 RGB or 16 YCrCb data formats, decoder block 225 is bypassed. 

Backend video pipeline 204 further includes a Y interpolator 226 and X 
25 interpolator 227. In the preferred embodiment, during Y zooming (expansion) Y 
interpolator 226 accepts two vertically adjacent 16-bit RGB or YCrCb pixels from 
the decoder 225 and calculates one or more resampled output pixels using a four 
subpixel granularity. X interpolator 227 during X zooming (expansion) accepts 
horizontally adjacent pixels from the Y interpolator 226 and calculates one or 
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more resampled output pixels using a four subpixel granularity. For data 
expansion using line replication, Y interpolator 226 is bypassed. Y interpolator 
226 and X interpolator 227 allow for the resizing of a video display window being 
generated from one to four times. 
5 The output of X interpolator 227 is passed to a color converter 228 which 

converts the YCrCb data into RGB data for delivery to output multiplexer 304. To 
reiterate, if graphics data is passed through the video pipeline, converter 228 is 
not used. 

Backend video circuitry 204 further includes pipeline control circuitry 
10 229, overlay control circuitry 230 and output multiplexer 231. Pipeline control 
circuitry 239 controls the reading of data from video FIFOs 223 and 217, controls 
vO the generation of interpolation coefficients for use by X and Y interpolators 226 

□ and 227 to resize the video window being pipelined, and times the transfer of data 

f| through the pipeline. Overlay control circuitry 230 along with control circuitry 

*P 15 202, controls the output of data through output multiplexer 231, including the 
i2 overlay of the video window over the graphics data output through the graphics 

= backend pipeline 205. A pixel doubler is provided to double the number of pixels 

p being generated such that a 1 280.times. 1 024 display can be driven. 

2 Graphics backend pipeline 205 includes a first-in/first-out memory 232, 

P 20 attribute controller 233, and color look-up table 234. Each 32-bit word output 
from graphics FIFO 232 is serialized into either 8-bit, 16-bit or 24-bit words. The 
8-bit words, typically composed of an ASCII code and an attribute code, are sent 
to attribute controller 233. When 16-bit and 24-bit words, which are typically 
color data, are serialized, those words are sent directly to overlay controls 230. 
25 Attribute controller 233 performs such tasks as blinking and underlining 
operations in text modes. The eight bits output from attribute controller 233 are 
pseudo-color pixels used to index CLUT 234. CLUT 234 preferably outputs 24- 
bit words of pixel data to output multiplexer 231 with each index. When video 
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data is being pipelined through graphics backend pipeline 205 from the on-screen 
memory, CLUT 234 is bypassed. 

The eight bit pseudo-color pixels output from attribute controller 233 are 
also sent to. overlay controls 230. In the preferred embodiment, data is 
5 continuously pipelined from on-screen memory through graphics backend 
pipeline 205 to the inputs of output multiplexer 231. Window data from off- 
screen memory however is only retrieved from memory and pipelined through 
video backend pipeline 204 when a window is being displayed. In other words, 
when a window has been reached, as determined by control bits set by CPU 101 
10 in VW control registers 222, video window display controls 222 generate 
addresses to retrieve the corresponding data from the off-screen memory space of 

£ frame buffer 1 07. Preferably, video FIFOs 223 and 224 are filled before the raster 

O 

□ scan actually reaches the display window such that the initial pixel data is 

^ available immediately once the window has been reached. In order to insure that 

*P 1 5 graphics memory data continues to be provided to graphics backend pipeline 205, 
y* video window display controls 222 "steal" page cycles between page accesses to 

' t ■ the graphics memory. It should be noted that once the window has been reached 

O the frequency of cycles used to retrieve window data increases over the number 

2 used to fill the video FIFOs when outside a window. When the frequency of 

C 20 window page accesses increases, video window display controls 222/arbiter 221 

preferably "steal" cycles from page cycles being used to write data into the frame 

buffer. 

FIG. 3 is a more detailed functional block diagram emphasizing the 
circuitry controlling the overlay of data from graphics pipeline 205 with window 
25 data from video pipeline 204. As discussed briefly above, the inputs to output 
multiplexer 231 are data from video backend pipeline 204 (pixel doubler 237), 16 
or 24-bit color data directly from graphics backend pipeline 205 serializer 236 and 
24-bit color data from the color look-up table 234. The output of data to DAC 210 
through output multiplexer 231 is controlled by a latch 301 clocked by the video 
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clock (VCLK). The remaining circuitry shown in FIG. 3, which will be discussed 
in further detail below, provide the necessary control signals to the control inputs 
of output multiplexer 231 to select between the video and graphics pipelines. 

The graphics pseudo-pixels output from attribute controller 233 and the 
16-bit or 24-bit graphics or video data output directly from serializer 236 are 
provided to the inputs of color comparison circuitry 302. Also input to color 
comparison circuitry 302 are 16 or 24-bit overlay color key bits stored in overlay 
color key register 303. Overlay color key register 303 resides within the address 
space of, and is loaded by, CPU 101. Depending on the mode, color comparison 
circuitry 302 compares selected bits from the overlay color key register 303 with 
either the 8 bits indexing look-up table 234 in the color look-up table mode 
(pseudo-color mode) or the 16-bits (24-bits in the alternate embodiment) passed 
directly from serializer 236. It should be noted that in the illustrated embodiment, 
overlay color key register 303 holds 24 overlay color key bits, eight each for red, 
green, and blue index comparisons. The specific overlay color key bits compared 
with the input graphics data are provided in Table I: 



OVERLAY COLOR KEY BITS COMPARED 



CLUT 
Index 



Blue/ Index 
<7 : 0> 



Red<4 : 
Red<4 : 
Red<7 : 



Green<4 : 0> 
Green<5 : 0> 
Green<7 : 0> 



As shown in FIG. 4A, a first embodiment of color comparison circuitry 
303 performs the comparisons set forth in Table I as a set of XNOR operations in 
series with an AND operation. FIG. 4A depicts first comparison circuitry 400 for 
comparing the 8-bits of graphics pixels received in the look-up table mode from 
attribute controller 233 with the 8-bit blue/index overlay key bits being held in 
overlay key register 303. Second comparison circuitry 401, performs the required 
comparisons of Table I for the 16-bit data or 24-bit received from serializer 236, 
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in either a 5:5:5, 5:6:5, or 8:8:8 format. An overlay register 402 includes a bit 
loaded by CPU 101 which is used by a selector 403, depending on the mode, to 
select for output, either the result of the comparisons being made by comparison 
circuitry 400 in the color look-up table mode or the results of the comparisons 
5 being made by comparison circuitry 401. In the illustrated embodiment, color 
comparison circuitry 303 processes data on a pixel-by-pixel basis and is 
resynchronized with both the graphics backend pipeline 205 and the video 
backend pipeline 204 by having its outputs latched to the video clock (VCLK) by 
latches 404. 

10 The output of color comparison circuitry 303 is passed to the "K" control 

input of overlay control multiplexer 304. The "P" control input to multiplexer 304 
*V 5 is provided from pixel position comparison circuitry 305. The data inputs to 

jjpvO multiplexer 304 are coupled to an 8-bit overlay OP Code (OOC) register 306. The 

\ Jp output of multiplexer 304 is used as one control input to output multiplexer 304, 

* -P 15 which along with a single bit set by CPU 101 into output control register 307, 
U. selects which of the data received at the data inputs of multiplexer 231 will be 

* u output to D AC 210. 

Q Pixel position comparison circuitry 305 includes three inputs coupled 

jj respectively to video window 1 position control circuitry 308, CRT position 

P 20 control circuitry 309 and video window 2 position control circuitry 310. In the 
illustrated embodiment, CRT position controller 309 is located within CRT 
controller 220 while video window 1 position control circuitry and video window 
2 position control circuitry 310 are located within video display window controls 
222 (FIG. 2). CRT position control circuitry 309 includes counters which track 
25 the position of the current pixel being generated for display. In the preferred 
embodiment, CRT position control circuitry 309 includes at least an x-position 
counter which tracks the generation of each pixel along a given display line and a 
y-position counter which tracks the generation of each display line in a screen. 
The x-position counter may for example count pixels by counting each VCLK 
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period between horizontal synchronization signal (HSYNC) controlling display 
unit 106. The y-position counter may for example count each HSYNC signal 
occurring between each vertical synchronization signal (VSYNC) controlling the 
screen generation on display unit 106. FIG. 4B is an alternate embodiment of the 
5 color comparison circuitry of FIG. 3. In a first mode, 8 bits from attribute 
controller 233 are passed through multiplexer 405 to comparator 406. Comparator 
406 compares the received eight bits with an 8-bit color key in color key register 
408; when the received 8-bits equal the 8-bit key 1, the output of comparator 406 
goes active (high). In the first mode, control signal 16BITGR is high (and the 




10 output of NOR gate 409 is consequently high) and an active output from 
comparator 406 is gated through AND gate 410. The output of AND gate 410 is 
passed to AND gate 411 and gated with the output from the pixel comparison 
circuitry 305. The output of AND gate 411 goes directly to the "B" control input 
of selector 231 (in this embodiment multiplexer 304 and register 306 are 



B F 15 eliminated). Thus, when the 8-bit graphics pixels output from attribute controller 
233 of graphics backend 205 matches the 8-bit color key 1 and the window has 
* u been reached as determined by pixel comparison circuitry 305, the pixel data 

O output from video backend 204 are passed through selector 23 1 . 

2 In a second mode, 1 6 bits are received from serializer 236. The eight LSBs 

P 20 are passed through multiplexer 405 to comparator 406 and the eight MSBs passed 
to comparator 407. Control signal 16BITNG is set high. When the LSBs equal 
key 1 in color register key 408 and the 8 MSBs equal key 2 in color key register 
408, the outputs from comparators 406 and 407 are active (high). The output of 
AND gate 411 then goes high when the output from pixel comparison circuitry 
25 305, which is coupled to the "B" control input of selector 231, goes high. Thus, 
when the 16-bit pixel data output from serializer 236 of graphics backend 205 
matches the 16-bit color key (keys 1 and 2) and a window has been reached, the 
output pixel data from video backend 204 are passed through selector 231. 
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FIG. 5 is an expanded functional block diagram of the video window 
position control circuits 308 and a corresponding portion of the gating of pixel 
position compare circuitry 305. Each position control circuit 310/312 is coupled 
to a screen position x-register 500 and a screen position y-register 501, and 
includes a screen x-position counter 502, and a screen y-position counter 503. In 
the preferred embodiment, registers 500 and 501 are located within video window 
control registers 203. For the window corresponding to the given video window 
control circuitry 308 or 310, registers 500 and 501 are loaded with a value 
representing the x and y screen position of the pixel in the upper left corner of that 
window (the starting pixel). Screen x-register 500 and screen y-register 501 in the 
preferred embodiment are loaded by CPU 101. The screen x-position counter 502 
counts down from the value held in screen x-register 500 with each video clock 
when P is high for each display line and resets with each display horizontal 
synchronization signal (HSYNC) (Note , that when P is high the CRT count 
matches the position count). Screen y-position counter 503 counts down from the 
value set into screen y-register 501 for each horizontal sync signal (HSYNC) at 
the start of each display line and resets with each VSYNC at the start of each new 
screen (The position counters are allowed to count only when they match their 
perspective CRT). The counts values in the counters of CRT position control 
circuitry 309 are compared pixel by pixel with the counts in screen x-position 
counter 502 and screen y-position 503 of each video window position control 
circuitry 308 and 310. When both the x and y counts in the counters of CRT 
position control circuitry 309 match the corresponding x and y counts in 
respective counters 502 and 503 of either video window control circuitry 308 or 
310, the control signal P to multiplexer 304 is activated. The activation of control 
signal P indicates that the raster scan on display 106 has reached the position of a 
pixel within the window and data from video pipeline 205 may be painted 
depending on the value being held in overlay OP Code (OOC) register 306 and 
the K control inputs to multiplexer 304. 
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A 4-bit OP Code loaded by CPU 101 into overlay OP Code register 306 in 
conjunction with the control signals applied to the "P" and "K" control inputs to 
multiplexer 304 control the presentation of an active (assumed high in the 
illustrated embodiment) control signal to the "B" control input to output 
multiplexer 231. The other ("A") input to output multiplexer 231 receives a bit 
from overlay mode register 402 (FIG. 4), as loaded by CPU 101. In the illustrated 
embodiment, the selection between the streams from the graphics and video 
backends at the 0,1,2 inputs to output multiplexer 304 in response to the signals 
presented at the corresponding control inputs "A" and "B" is in accordance with 
Table II: 



Control Input A 

Control Input B 

Selected Stream 



Graphics or 
video pixels 
from graphics 
pipeline 205 
Graphics pixels 
from CLUT 234 at 
input 1 
Video or 
graphics from 
video backend 
204 



The OP Codes used in the illustrated embodiment, the effective overlay 
and the corresponding inputs to the control inputs of multiplexer 304 are listed in 
Table III (active state is assumed): 
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Overlay Op Code 

Multiplexer 307 
Register 309 Value 
5 Control Inputs 

Effect 

0 N/A Pixels passed 

only from 

10 graphics pipeline 

205 

A Input P active 

Paint pixels from 
video backend 
15 pipeline 204 only 

inside video 
window VDW.sub.n 
8 Inputs K and P 

Paint from video 
Id 20 active backend pipeline 

Q 2 04 window VDW.sub.n 

when color key 
matches 

C Inputs K active 

25 Paint from video 

backend pipeline 
204 if color key 
matches 



In the illustrated embodiment, if a Oh is written into OOC register 306 by 
CPU 101, only pixels from graphics pipeline 205 are pipelined through 
multiplexer 304. In this case any signals applied to the P and K control inputs to 
multiplexer 304 have no effect (i.e., will not result in a high output from 
multiplexer 304). In the illustrated embodiment, if an Ah is written into OOC 
register 306, pixels from video pipeline 204 will be passed to DAC 210 only 
when pixel position comparison circuitry 305 determines that the raster scan has 
reached a pixel in the window and hence the control signal going to the P input of 
multiplexer 304 has been activated. If on the other hand, an 8h is written into 
OOC register 306, data is passed through output multiplexer 231 to DAC 210 
when pixel position comparison circuitry 305 determines that the raster scan has 
reached a pixel on the display screen within the window and color compare 
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circuitry 302 has determined that the incoming data from graphics pipeline 205 
matches the overlay color key held in overlay color key register 303. In this case, 
the data from video pipeline 204 is passed to DAC 210 when both the P and the K 
inputs to multiplexer 304 are active. Finally, when an OpCode of C is 
programmed into OOC register 306, data from video pipeline 204 is passed if the 
incoming data from graphics pipeline 205 matches the overlay color key held in 
overlay color key register 303. In this case, the activation of the K control input 
activate the output of multiplexer 304 to switch the input of multiplexer 231 to 
pass the corresponding video pixels. 

Display control circuits embodying the principles of the present invention 
have substantial advantages over the prior art. In particular, output control circuits 
built in accordance with the principles of the present invention allow for the 
flexible display of both graphics and video on the same screen. In particular, pixel 
position comparison circuitry 305 along with video window position control 
circuits 308 and 310 and CRT position control circuitry 309 allow for one or more 
windows from off-screen memory to be generated in specific areas of a display 
screen to the exclusion of any simultaneously generated data from on-screen 
□ memory. Further, color comparison circuitry 302 operating in conjunction with an 

overlay color key written into overlay color key register 303 allows window data 
to be presented on the display screen, to the exclusion of any concurrently 
generated graphics data, without the need for precise x- and y-position data for the 
window. Finally, the use of the graphics data from the graphics pipeline 205 to 
control the output overlay provides additional advantages since the video data can 
be subject to graininess and noise. 

Although the present invention and its advantages have been described in 
detail, it should be understood that various changes, substitutions and alterations 
can be made herein without departing from the spirit and scope of the invention as 
defined by the appended claims. 
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